Many opportunities, and a few challenges
March 10, 2025
In divergent thinking tasks, fluency scores are the count of unique responses provided by a respondent.
They come with unique psychometric challenges…
…item response theory allows these flexibilities !
According to the 2PPCM (Myszkowski & Storme, 2021), the fluency score \(X_{ij}\) of person \(i\) for item \(j\) comes from a Poisson distribution of rate parameter \(\lambda_{ij}\)…
\[X_{ij} \sim \text{Poisson}(e^{a_j\theta_i + b_j})\]
Primarily developed through maximum likelihood approaches
Mplus, Stata)
countirt)
lme4)
Can a general purpose Bayesian estimation framework do better?
Is it feasible to estimate log-linear count IRT models like the 2PPCM in a Bayesian framework with packages non-dedicated to count IRT?
Can we obtain results similar to maximum likelihood estimates by using weakly/non-informative priors?
Are the benefits of count IRT models limited or expanded by a Bayesian framework?
Are the benefits of a Bayesian framework (reasonably) attainable? Are they (possibly) useful in the context of divergent thinking tests?
Are there outstanding issues? New areas of development?
Stan and brmsStan (Carpenter et al., 2017): A programming language suited for Bayesian using Hamiltonian Monte Carlo (HMC) estimation.
brms (Bürkner, 2017): An R package to estimate various models in Stan using regression-like syntax (e.g., y ~ x1 + x2)
Stan syntaxIt’s not too bad ! See our paper (Myszkowski & Storme, 2025)…
…but here’s a quick look !
Publicly available dataset for special issue (Forthmann et al., 2019)
202 respondents (variable Person)
3 alternate uses tasks (rope, paperclip, garbage bag) (variable Item)
Main analysis: 2PPCM with random person effect and fixed item effects (identified with variance standardization method)
brmsformula_2PPCM <- bf(
Score ~ 0 + slope * theta + easiness, #linear part of the item response model
theta ~ 0 + (1 | Person), #Theta is a random effect of the person
slope ~ 0 + Item, #Slope is a fixed effect of the item
easiness ~ 0 + Item, #Easiness is a fixed effect of the item
nl = TRUE, #Tells brms that we are doing weird non-linear things
family = poisson(link = "log") #Log-linear Poisson model
)Full posterior distributions of item parameters and ability (as opposed to points with standard errors).
“Is item 1 easier than item 3?”
“Is person 1’s fluency more than 1 standard deviation higher than person 2’s fluency?”
Treat item characteristics (e.g. slope) as random deviations from a shared distribution.
Item covariates (e.g., object type):
formula_2PPCM_expl <- bf(
Score ~ 0 + slope * theta + easiness,
theta ~ 0 + (1 | Person),
slope ~ 0 + Item,
easiness ~ 0 + object_type,
nl = TRUE,
family = poisson(link = "log")
)Person covariates (i.e. latent regression/latent mean differences)
In maximum likelihood estimation, items with very high or low discrimination can cause instability in the model (especially in small samples).
By using informative priors (i.e. Bayesian regularization), we can avoid unrealistic parameter values and unstable models.
Applicable to extensions (e.g., avoiding differential item functionning false positives)
Rather than select one model with a set of priors, we can combine multiple models and/or multiple sets of priors using Bayesian averaging.
For example, we can obtain \(\theta\) posterior distributions from different models, which we average in proportion to the posterior model probabilities.
Avoids reliance on a single model, leading to more robust predictions.
Default priors were sufficient for the RPCM, but not for the 2PPCM.
Find this presentation at https://osf.io/9f4eu/